Construct and Criterion-related validation of the Bifactor Engagement Scale
John Kulas Mike DeFabiis Casey Osorio Morgan Russell
# Seed for random number generation
set.seed(42)
knitr::opts_chunk$set(cache.extra = knitr::rand_seed, warning=FALSE, echo=FALSE)The roots of employee (sometimes aka work, e.g., Schaufeli & Bakker, 2010a) engagement research likely started with theoretical expansions of forms of employee participation (see, for example, Ferris & Hellier, 1984) and job involvement (e.g., Elloy et al., 1991). This exploration extended into broader considerations of attitudes and emotions (Staw et al., 1994) and were informed by further exploration of the dimensionality of constructs such as organizational commitment (Meyer & Allen, 1991). The 1990’s saw focused development and refinement. Staw et al. (1994) investigated the relationships between positive emotions and favorable work outcomes, and although they do not use the word, “engagement”, their distinction between felt and expressed emotion likely held influence upon the burgeoning interest in the engagement construct.
Kahn (1990) described engaged employees as being physically involved, cognitively vigilant, and emotionally connected. Although occasionally referred to as residing on the opposing pole to burnout (Maslach & Leiter, 2008), these two constructs are currently most commonly conceptualized as being distinct (Goering et al., 2017; Kim et al., 2009; Schaufeli et al., 2008; Timms et al., 2012), although certainly not universally (Cole et al., 2012; Taris et al., 2017). Goering et al. (2017) explore nomological networks, concluding that these two constructs have a moderate (negative) association, but also distinct nomological networks. Schaufeli et al. (2008) investigated both internal and external association indicators, concluding that engagement and burnout (as well as workaholism) should be considered three distinct constructs.
Burnout can be defined as a psychological syndrome characterized by exhaustion (low energy), cynicism (low involvement), and inefficacy (low efficacy), which is experienced in response to chronic job stressors (e.g., Leiter & Maslach, 2004; Maslach & Leiter, 1997). Alternatively, engagement refers to an individual worker’s involvement and satisfaction as well as enthusiasm for work (Harter et al., 2002).
Engagement as an attitude
Staw et al. (1994) investigated the relationships between positive emotions and favorable work outcomes, and, although they do not explicitly mention the word “engagement”, their distinction between felt and expressed emotion likely held influence upon the burgeoning interest in the engagement construct. Clear in this history is the conceptualization of engagement as a work attitude. Kahn (1990) defines engagement as “the harnessing of organization members’ selves to their work roles; in engagement, people employ and express themselves physically, cognitively, and emotionally during role performances” (p. 692). This definition of engagement as an attitude was also heavily influenced by Rosenberg (1960)’s tripartite model of attitudes, which was popular in the 1990’s. According to Rosenberg (1960), attitudes are a molar construct with cognitive, affective, and behavioral dimensions. Although falling out of favor in the decades following its construction, interest in the tripartite model was revived by Kaiser & Wilson (2019). The attitudinal perspectives of engagement eventually blended into perspectives that focused on exploring the engagement construct through the lens of other conceptually similar constructs Shaw (2005).
Existing Measures of Engagement
Our review of existing instruments non-exhaustively presents measures that are commonly viewed as either predominantly academic or applied, although please note that this is an imposed subjective distinction.
Research measures (e.g., freely available).
Schaufeli & Bakker (2003) characterize engagement as a “positive, fulfilling, work-related state of mind that is characterized by vigor, dedication, and absorption” (p. 74). Via their conceptualization, vigor is described as high levels of energy and mental resilience while working. Dedication refers to being strongly involved in one’s work and experiencing a sense of significance, enthusiasm, inspiration, pride, and challenge. Absorption is characterized by being fully concentrated and happily engrossed in one’s work, whereby time passes quickly and one has difficulties with detaching oneself from work (Schaufeli et al., 2002). This absorption element has been noted as being influenced in conceptual specification by (Csikszentmihalyi, 1990)’s concept of “flow”. Schaufeli & Bakker (2003) use this tripartite framework to measure engagement via the Utrecht Work Engagement Scale (UWES).
The Intellectual, Social, Affective (ISA) Engagement Scale (Soane et al., 2012) is another option for researchers. This 9-item measure draws inspiration from Kahn (1990)’s theory of engagement and can aggregate to three 3-item scales (Intellectual Engagement, Social Engagement, and Affective Engagement) or one 9-item summary aggregate (Overall Engagement). Intellectual engagement refers to the degree of intellectual absorption one has in their work and the degree they think about improving work (Soane et al., 2012). Social engagement primarily concerns social connections in a workplace context as well as having shared values with colleagues (Soane et al., 2012). According to Soane et al. (2012), affective engagement refers to a positive emotional state relating to one’s work role. This measure has been explicitly validated at both the subscale and overall aggregate level (Soane et al., 2012).
Another example of an engagement measure comes from Saks (2006), who splits engagement into two distinct entities: job engagement and organization engagement. This dichotomy largely results from Kahn (1990)’s theory that an individual’s role is central to engagement. Saks (2006) further posits that employees typically have more than one role, with the most important being their work role and their role as a member of an organization. The former role is specific to the employee’s job, while the latter is more broad and refers to the organization as a whole. Antecedents and consequences of this measure have been tested, with findings suggesting that perceived organizational support precedes both job and organizational engagement and that job satisfaction, organizational commitment, intent to quit, and organizational citizenship behaviors (OCBs) are consequences (Saks, 2006). Recently the broader theoretical model underpinning the measure was revisited and revised to include several new antecedents (e.g. leadership, job demands, dispositional characteristics, etc.) leading to engagement as well as consequences (e.g. burnout, stress, health and well-being, etc.) resulting from high or low levels of engagement (Saks, 2019).
Commercial measures (e.g., typically fee-based).
Gallup’s Q12 is a popular commercial measure for engagement. The Q12 is a 12-item measure that originated from a push to use “soft” metrics as opposed to “hard” ones for future action planning (Coffman & Harter, 1999). In this interpretation “soft” metrics tend to be metrics that are more abstract and difficult to measure (e.g. engagement, brand loyalty), while “hard” metrics are easily-measured and typically deal with concrete numbers (e.g. turnover, profitability). In the original creation of the survey, each of the 12 items were found to relate to important organizational outcomes including productivity, profitability, turnover, and customer satisfaction (Coffman & Harter, 1999). A recent meta-analysis of 456 studies revealed that the Q12 also relates to additional performance measures such as absenteeism, wellbeing, and organizational citizenship (Harter et al., 2013). While this engagement measure is one of the most popular, some scholars disagree with its conceptualization as “engagement”; some feel that this measure is better described as (or no different than) a measure of overall satisfaction, as the two concepts are highly correlated, r = .91 (Sirota & Klein, 2013).
Gallup is not the only organization with an engagement measure; many consulting companies have commercially available surveys, models, and processes for measuring engagement. One such example is Aon Hewitt, a consulting firm that annually measures engagement for over 1000 companies worldwide. Their measurements are centered around an engagement model that focuses on three main factors: say, stay, and strive. Essentially, the model states that employees demonstrate engagement through saying positive statements about their organization, staying at their organization for a long time, and striving to put in their best effort and help the organization succeed (Hewitt, 2017). In their most recent analysis. Hewitt (2017) recently noted that global levels of engagement may be declining as in this report they had retracted since the previous year.
BlessingWhite, another consulting firm, provides a different model for engagement. BlessingWhite’s model, the X Model, measures engagement through the lens of satisfaction and contribution. Essentially, BlessingWhite believes that cooperation between the organization and individual employees is necessary, and that maximum engagement can only be reached when an employee reaches maximum levels of satisfaction while also outputting maximum contribution towards the organization (BlessingWhite, 2018). Their model holds each level in the organization accountable for employee levels of engagement. From their view, executive leaders must shape the organization’s culture, and managers must be able to effectively communicate with and motivate their subordinates (BlessingWhite, 2018).
The last commercial example discussed here1 is the Towers Perrin-ISR, which holds the philosophy that employee engagement can only be worked on indirectly; engagement can only be attained through effective leadership, business strategy, and organizational culture (Ballendowitsch & Perrin-ISR, 2009). Rather than focus on building an involved model for engagement, Towers Perrin-ISR instead focuses on leadership development and creating a healthy organizational culture. Through fulfilling these antecedents of engagement, Ballendowitsch & Perrin-ISR (2009) argues that employees will have a vivid understanding of organizational goals. In addition, employees will become committed to the organization and motivated to contribute.
Our Measure of Engagement
Our theoretical conceptualization of work engagement is primarily informed by Schaufeli & Bakker (2003) and Rosenberg (1960). Through the lens of our framework, engagement is a mental state wherein employees: a) feel energized (Vigor), b) are enthusiastic about the content of their work and the things they do (Dedication), and c) are so immersed in their work activities that time seems compressed (Absorption). We further decompose each of these facets into three attitudinal components: d) feeling (e.g., affect), e) thought (e.g., cognition), and f) action (e.g., behavior).
Schaufeli (2013) stated a preference for the label “work engagement” rather than referring to the construct as “employee engagement”, arguing that the “employee” referrent perhaps invites a blurring of definitions with other conceptually similar constructs such as commitment or organizational citizenship. Regarding this distinction between “the job” and “the organization”, our measure scatters indicators of both throughout, although we did not intentionally balance the measure with regard to the referent, as do others, such as Saks (2006).
The current study’s focus is on exploring external variable associations with our measure, focusing on indicies of construct and criterion-related validity via retention of two alternative measures of engagement (the Saks scale and the UWES), two measures of theoretically orthogonal constructs (activity regarding household chores and tending to pets), and one measure of a theoretically relevant outcome (intentions to quit).
Methods
We purchased Qualtrics panels of working adults and administered a standard Qualtrics survey via online delivery, however, as noted below, very cautious screening for indicators of careless responding resulted in our exclusion of many of these Qualtrics respondents from our presented analyses. The total survey was comprised of 74 items across 6 constructs of interest as well as several demographic items that are not the focus of this current presentation.
Participants
Of the 743 total Qualtrics panel respondents, roughly half were excluded based on conservative indices of carelessness across the larger survey. These screens included respondents with more than 50% missing responses, those who provided consistently non-differentiating responses across more than 12 consecutive items, and those who completed the survey in less than 300 seconds. These conservative screens resulted in a retained validation sample of 377. All analyses were derived from this n of 377.
Data analysis
We used R (Version 4.2.1; R Core Team, 2022) and the R-packages papaja (Version 0.1.1; Aust & Barth, 2022), psych (Version 2.2.5; Revelle, 2022), and tinylabels (Version 0.2.3; Barth, 2022) for all our analyses. As a straightforward validation study, our analyses consisted predominantly of Pearsons product-moment correlations.
Results
Suggested final scale definitions.
| Substantive | Attitudinal | Item.Number | Item.Stem |
|---|---|---|---|
| Absorption | Cognitive | 1 | I am able to concentrate on my work without getting distracted |
| Absorption | Cognitive | 3 | Time passes quickly while I’m working |
| Absorption | Affective | 5 | I enjoy thinking about work even when I’m not at work |
| Absorption | Affective | 8 | I love starting my workday |
| Absorption | Behavioral | 10 | I have to be reminded to take breaks while I’m at work |
| Absorption | Behavioral | 11 | I never miss a work deadline |
| Vigor | Cognitive | 14 | Thinking about work saps my energy |
| Vigor | Cognitive | 16 | I’m able to maintain good levels of energy throughout the workday |
| Vigor | Affective | 17 | I enjoy spending time completing my job tasks |
| Vigor | Affective | 19 | I feek motivated to go beyond what is asked of me at work |
| Vigor | Behavioral | 21 | When work is slow I find ways to be productive |
| Vigor | Behavioral | 22 | I express enthusiasm for my job while at work |
| Dedication | Cognitive | 25 | I plan to stay with this company as my career advances |
| Dedication | Cognitive | 26 | I believe this company cares about my career goals |
| Dedication | Affective | 31 | I feel proud of my accomplishments within this organization |
| Dedication | Affective | 32 | My job makes me feel like I’m part of something meaningful |
| Dedication | Behavioral | 34 | I embrace challenging situations at work |
| Dedication | Behavioral | 35 | I speak positively about this organization to others |
Note. The recommended response scale is ‘Strongly Disagree’, ‘Disagree’, ‘Somewhat Disagree’, ‘Somewhat Agree’, ‘Agree’, and ‘Strongly Agree’
The items comprising the focal measure along with their scale associations and recommended administered response scale are located in Table 1. The current sample internal consistency estimates for our three substantive subscales were: 1) Absorption (\(\alpha\) = 0.75), 2) Dedication (\(\alpha\) = 0.89), and 3) Vigor (\(\alpha\) = 0.75), and estimates for our three attitudinal subscales were: 1) Affect/“Feel” (\(\alpha\) = 0.86), 2) Behavior/“Do” (\(\alpha\) = 0.77), and 3) Cognition/“Think” (\(\alpha\) = 0.77).
Construct validation
For convergent validity indices, we administered the 17-item Utrecht Work Engagement Scale (Schaufeli et al., 2002; Schaufeli & Bakker, 2010b) as well as Saks (2006)’s 12-item measure which aggregates to two scales: job and organizational engagement (see also Saks, 2019).2 An example item from the Saks (2006) (job) scale is, “Sometimes I am so into my job that I lose track of time”. An example item from the Schaufeli et al. (2002) scale is, “At my work, I feel bursting with energy”. The Schaufeli et al. (2002) measure follows the same structure as our focal measure, so we aggregated to subscales of Absorption (\(\alpha\) = 0.84), Dedication (\(\alpha\) = 0.87), and Vigor (\(\alpha\) = 0.85). Internal consistency estimates for the Saks scale were \(\alpha\) = 0.69 (job engagement) and \(\alpha\) = 0.84 (organizational engagement). Also note here that the English version of the UWES may actually be a translation (it is difficult to say for sure, as the test manual describes an original Dutch sample although the manual is written in English). Further suggesting that the English version may be a translation, some items have odd grammar (for example, “I am proud on [sic] the work that I do”).
Two short scales from the Oregon Avocational Interest Scales (Goldberg, 2010) were retained for discriminant validitation - the 5-item “Pets” and 5-item “Household Activities” scales. These items asked how frequently respondents engaged in different activities. An example Household Activity item is, “Cleaned the house” (current sample \(\alpha\) = 0.72) and an example Pets item is “Fed a pet animal” (current sample \(\alpha\) = 0.88).
Discussion
Our primary aspiration for developing this measure was that it would be a public domain instrument that would draw equal appeal from both practitioners and academics. These preliminary investigations suggest that it is scaleable to two aggregations which we have been referring to as: 1) research (DAC), and 2) actionable (ABC). Our (as-of-yet untested) assumption is that practitioners may be more interested in feedback regarding how their employees think, feel, and behave with regard to engagement. Academics, on the other hand, may be more interested in possible differentiation between levels of dedication, absorption, and vigor. Having one assessment that may aggregate to either framework not only addresses the demand of constituent users, but it also facilitates aggregation across samplings for broader purposes such as norms development, validation, and metaanalysis.
The convergent indices provide preliminary evidence that the three engagement measures are measuring similar but not redundant content. The criterion-related indices suggest that the focal variable may have superior prediction, although more validation needs to occur, both with turnover intentions as well as actual turnover behavior. The disciminant validity indices did exhibit magnitudes slightly higher than anticipated, however, upon close inspection, the largest coefficients (r’s of .15 and .17) emerged across the focal “behavior” and “dedication” scales. In retrospect, our sample respondents who engaged in more engagement behavior and exhibited higher levels of dedication could very well be expected to also extend those proclivities beyond work - perhaps including household and pet-care activities.
Although not explored here, there is also further predictive power potentially located within our intentionally complex instrument. It is possible that combined scale focus (for example, “Cognitively Dedicated” - shared items across the cognitive and dedication scales) exhibits even more predictive power for targeted outcomes of interest. Future investigations may wish to additionally probe for associations at this “cell” level.
References
This non-exhaustive list is not meant to be comprehensive. We intended to present some popular measures (albeit from larger vendors) in an attempt to capture the variety of rationales and purposes behind the creation and administration of these measures.↩︎
We had also intended to use the Gallup “Q12” for construct validation (Harter et al., 2013; Thackray, 2005), but Gallup was not willing to share item- or person-level data.↩︎